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ABSTRACT 


In this paper, we describe the impact of two interventions in the design of mail surveys. The interventions 
were devised to increase response rates and to clarify sample eligibility. To test their effectiveness, the 
interventions occurred at different points in each of three surveys. One intervention was a letter from the research 
sponsor (the National Aeronautics and Space Administration) supporting the research. The other intervention was 
the inclusion of a postcard that could be used by the respondent to indicate that the questionnaire was not 
appropriate for him/her. 

The sample was drawn from the membership of a professional aerospace research society - the American 
Institute for Aeronautics and Astronautics. Scientists and engineers are difficult to survey for two reasons. First, 
there are significant problems with the definition of scientists and engineers (Citro and Kalton, 1989). Second, 
typically there are low response rates in surveys of this group (Citro and Kalton, 1989). These two problems were 
found in the NASA surveys. 

The results indicate that the sponsor letter improved response rates under certain conditions described in 
the paper. The postcards assisted in identifying non-eligible persons particularly when they accompanied a pre- 
survey letter. The implications for survey costs are discussed. 



The Impact of a Sponsor Letter on Mail Survey Response Rates 

Survey researchers continually search for techniques to improve response rates and reduce 
survey costs and errors In this paper, we describe how the timing of a letter from a sponsoring 
government agency affected the return rates in three mail surveys. In addition, we describe how a 
postcard was utilized to clarify the sample in the surveys. Together, these procedures allowed us to assess 
the relative impact of survey design changes on response rates and survey costs. 

Our research was prompted by the need to refine our mail survey procedures for repeated 
samples from a specific target population - domestic and international aerospace scientists and engineers. 
In the past year, the Center for Survey Research at Indiana University has conducted six surveys for the 
National Aeronautics and Space Administration (NASA). In this paper we present some of the results from 
three of the surveys which were similar in content and target population. The focus of the research is the 
analysis of the transfer of scientific and technical information (STI) from research labs, such as NASA’s, to 
engineers and scientists in the aerospace industry and in universities. In the initial phase of the research 
project, three mail surveys of aerospace scientists and engineers were conducted to determine their use of 
a variety of information sources that might be used in aerospace research. Subsequent phases of the 
project would involve surveying additional domestic and international aerospace scientists and engineers. 

We began our research project using the Total Design Methods described by Dillman (1978). In 
two prior surveys of professionals conducted at the Center for Survey Research these procedures 
generated response rates of 85 percent (university faculty) and 73 percent (lawyers in Indiana). The first 
two surveys in the NASA project generated response rates of less than 60 percent. Since additional 
surveys were proposed as part of the project, we decided to experiment with alternative survey procedures 
to increase response rates. The procedures tried were: 1) sending a sponsor letter from a NASA official to 
the non-respondents in the first two surveys; 2) sending a sponsor letter as a presurvey letter in the third 
survey; and, 3) including a postage-paid return postcard with the sponsor letters that would allow us to 
define more clearly the sample. In addition, we hoped that the results of these changes would allow us to 
reduce the costs of the future surveys. 
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Groves (1989; p. 49) argues that costs should be considered as a factor in survey design. As part 
of our research, we make some preliminary estimates of the costs of implementing the various design 
features. Our goals in the research were to assess the effect of the design changes on: 1) response rates; 
2) clarifying the appropriate sample; 3) the relative costs of each design; and 4) the demographic 
composition of the respondents. 

Design Effects on Mail Survey Response Rates 

In this paper, we examine the impact of four factors that have been found to increase response 
rates to mail surveys - viz, university sponsorship, government sponsorship, number of contacts, and 
presurvey letters. Fox et al. (1988) conducted a meta-analysis of nine factors thought to be related to 
improving response rates to mail surveys. In their research, they analyzed 148 articles that described 
various mail survey design procedures. Among the factors they found significantly related to response 
rates were university sponsorship and prenotification letters. 

Earlier, Heberlein and Baumgartner (1978, 1981) analyzed the impact of 71 variables used in 98 
mail survey experiments on final response rates to mail surveys. They found that ten variables including 
government sponsorship and the total number of contacts, could predict response rates. Goyder (1982) 
replicated the research of Heberlein and Baumgartner and found essentially the same results. Eichner and 

Habermehl (1981) were not able to replicate the results in their analysis of mail surveys conducted in 
Europe. 

The previous research has demonstrated that response rates are affected by a variety of 
characteristics of the survey procedures. None, though, analyzed the results in terms of the relative costs 
of the procedures and the Improvements in response rates. Also, the research cited has not focused on 
the possibility that design changes could differentially alter the response rates among various demographic 
groups in the sample. We suggest that procedures designed to improve response rates should 
simultaneously recognize the costs of the design and the potential of differential impact on non-response. 
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Surveying Engineers and Scientists 

Typically there are Igw response rates in surveys of scientists and engineers (Citro and Kalton, 
1989). One of the reasons often given to explain why they are difficult to survey is because of the problems 
in defining who is a scientist or engineer (Citro and Kalton, 1989). We expected that achieving high 
response rates would be a challenge because of the definition problem and because the topic of the 
questionnaires was not relevant to the professional activities of some unknown proportion of the sample. 

The focus of the surveys was to evaluate the information-seeking habits and practices of 
researchers in aerospace. A sample of aerospace engineers and scientists was chosen from the 
membership of the American Institute of Aeronautics and Astronautics (AIAA). Most researchers in 
aerospace belong to the AIAA because it is their primary professional research society. But, many 
members of AIAA are no longer involved in aerospace research and some are no longer in the aerospace 
industry. Therefore, many of the members of the sample could not be classified as aerospace research 
engineers or scientists. 

Further, we assume that questionnaires were sent to some AIAA members for whom the questions 
had no relevance. That is, many in the sample were not actively engaged in aerospace research in their 
professional duties. Therefore, we anticipated that the response rate would decrease because the salience 
of the topic was expected to be low for many in the sample (Heberlein and Baumgartner, 1978). 

The problems were further compounded by the historically low response rates in surveys of 
scientists and engineers. For example, the National Science Foundation conducted a survey of scientists 
and engineers in 1982. In this survey, the sample was mailed a questionnaire, a postcard, and up to two 
additional questionnaires were sent to the non-respondents. After the third questionnaire, a telephone 
follow-up was attempted with the remaining non-respondents. The response rate for the survey was 71 
percent. (This description of the survey came from Citro and Kalton; pg 67.) Citro and Kalton (1989; pg 
120) present additional evidence to show that response rates from this group have decreased during the 


1980's. 
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Survey Procedures 

The first two NASA surveys were conducted following the procedures described in Dillman (1978). 
In each survey, we mailed a questionnaire, a postcard, and a second questionnaire to the non- 
respondents. In the first survey, we mailed a third questionnaire to the non-respondents. From the first 
survey, we received about 58 percent usable questionnaires. In addition, about four percent of the sample 
returned questionnaires that were not usable (see Table 1). The second NASA survey was staggered one 
mailing behind the first survey and after mailing two questionnaires in the second survey, the proportion of 
usable questionnaires was about 50 percent. An additional five percent replied but did not provide a 
usable questionnaire. 

In this paper, three terms are used to describe the returns from the surveys. The "response rate’ is 
calculated on the number of usable responses divided by the total sample (RR = Usable/N). The "adjusted 
response rate" is the number of usable response divided by the sample minus the self-defined ineligible 
persons in the sample (ARR = Usable/[N - ineligible]). Respondents identified themselves as ineligible 
primarily because they were not engaged in aerospace research. In Table 1 , the "other" category refers to 
persons who were retired, moved to Europe, deceased, or otherwise ineligible for the survey. The "return 
rate" is the number of sample persons who responded in any form divided by the sample size. Comparing 
the response rate against both the adjusted response rate and the return rate is valuable in determining the 
relative costs of design changes. 

The results in Table 1 demonstrate the relatively low response rates in the two surveys. The 
proportion of usable responses was much less than had been our experience so we decided to continue 
the survey, primarily to increase the response rate but also to determine reasons for non-response and to 
improve the design for future surveys. 
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Table 1 : Detailed Response Outcomes from Two Mail Surveys. 


OUTCOMES 

NASA1 1 

NASA2' 

Usable 

57.8 

50.0 

Refused 

0.8 

0.6 

Not in aerospace 

1.0 

1.0 

Other reason 

1.9 

3.6 

Response Rate 

58% 

50% 

Adjusted Response Rate 

59% 

52% 

Return Rate 

61% 

55% 


1 . Initial mailing and postcard sent to all respondents; up to two additional questionnaires sent to non- 
respondents. 

2. Initial mailing and postcard sent to all respondents; one additional questionnaire sent to non- 
respondents. 


The Implementation of Additional Survey Design Features 

Two additional procedures were undertaken to increase the response rate and to establish the 
relevance of the survey focus for the remaining persons in the samples. First, a letter from a NASA official 
was sent to the remaining non-respondents. The letter explained the importance of the survey and 
requested cooperation with the project. All aerospace researchers are familiar with NASA, so it was 
assumed that a letter from a NASA Division Director would induce some of the non-respondents to 
complete and return the questionnaire. The impact of a letter from a government sponsor was found 
previously to significantly increase response rates by Heberlein and Baumgartner (1978). 

In order to identify the relevance of the content of the questionnaires, a return postcard was 
enclosed with the letter. This postcard allowed the recipient to check-off; 

1 . Please send me another questionnaire 

2. I am not involved in aerospace research 

3. Ido not wish to participate in the study 

4. Please send me a copy of the final report 
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The postcard was designed to help determine if the reason for the previous non-response was due 
to the content of the questionnaire. It was expected that this procedure would allow us to better determine 
the proportion of the sample that did not respond because the survey focus was not appropriate to them. 

At the same time as the sponsor letters were sent, the third survey began. Prior to mailing the 
questionnaires, a presurvey letter from a NASA official was sent. This letter was similar to the follow-up 
letter in the other surveys. Also included was a postcard that allowed the respondent to indicate that s/he 
is not involved in aerospace research. The postcard was similar to that sent in the other surveys except 
that it did not have a check-off to send another questionnaire. Two weeks were allowed for the 
respondents to return the postcard. At that time, all those who had not returned the postcard were sent 
questionnaires. A follow-up postcard and a second questionnaire to non-respondents were also sent. 


Table 2: Detailed Response Outcomes from Three Mail Surveys; Final Results. 


OUTCOMES 

NASA1 1 

NASA2 2 

NASA3' 

Usable 

61.4 

55.6 

54.1 

Refused 

4.4 

5.3 

4.5 

Not in aerospace 

6.0 

7.5 

13.1 

Other reason 

3.3 

3.7 

2.6 

Response Rate 

61% 

56% 

54% 

Adjusted Response Rate 

67% 

63% 

64% 

Return Rate 

75% 

72% 

74% 


1 . Initial mailing and postcard sent to all respondents; up to two additional questionnaires sent to non- 
respondents, sponsor letter sent last. 

2. Initial mailing and postcard sent to all respondents; one additional questionnaire sent to non- 
respondents, sponsor letter sent last. 

3. Presurvey sponsor letter sent to all, mailing and postcard to all who did not indicate not applicable, 
one additional questionnaire sent to non-respondents. 


The differences between Tables 1 and 2 indicate that the sponsor letter did not dramatically 
increase response rates. The Response Rate in the first and second surveys increased by about 4 and 6 
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percentage points respectively. A comparison of the third survey with the results of the second survey 
prior to the sponsor letter indicates that the sponsor letter may have increased the Response Rate in the 
third survey by about 4 percentage points. The response rates appear to be in the range currently 
experienced in surveys of engineers and scientists (Citro and Kalton, 1989). Since both government- 
sponsored and university-sponsored surveys were found to increase response rates to mail surveys, it is 
not surprising that the net effect of the sponsor letter was small. 

There is some evidence that the postcard was effective in focusing the sample. The number of 
persons who indicated the survey was not appropriate increased substantially after the postcard was sent 
in the first two surveys. The Adjusted Return Rates increased by 8 and 1 1 percentage points respectively In 
the two surveys that received the postcard in the last mailing. In the third survey about 13 percent 
indicated the topic was not relevant. About 70 percent of these persons indicated so prior to the first 
mailing. Overall, it seems that with further refinement of the initial letter and the postcard that this 
procedure can be used to refine samples where there is a problem with eligibility for the survey. 

Some further cautions. It is possible that at least three other factors differentially affected the response 
rates. First, the form of the questions and layout of the questionnaires was similar in each survey but the 
topics in each was a little different. In the first survey, the questions focused on a variety of government 
and nongovernment information sources. In the second survey, the questions focused on the use and 
availability of selected government and foreign technical reports. In the third survey, bibliographic sources 
of information about technical reports were researched. The data from the third survey indicate that very 
few of the respondents were familiar with most of the bibliographic sources and we suspect this lack of 
knowledge influenced the response rate. Since the results are similar across surveys, if these factors 
affected response, they probably did so independent of the letter. 

Second, the questionnaires in each survey were different colors. The first survey questionnaires 
were blue or green. Both colors had about equal response rates. The second questionnaire was a light 
brown and the third questionnaire was white. Fox et al (1988) found significant increases in response rates 
associated with green vs white questionnaires. Another possible factor that affected response rates was 
the time each survey was in the field. The first survey started in early summer, the second started in early 
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July, and the third survey began in early fall. We suspect that summer vacations and household migration 
affected the number of returns. 

The results from these surveys and design experiments along with the results from other surveys of 
scientists and engineers indicate that it would be difficult to increase response rates above the mid-60 
percent levels. Therefore, the procedures for future mail surveys can be changed to reflect the relative 
costs and the overall impact on nonresponse bias. That is. if there are substantial differences in the costs 
of each set of procedures, it would appear to be best to use the least costly design so that more 
questionnaires could be returned. Further, if a procedure affects nonresponse bias (see Groves, 1989; pg 
40-41 for a definition of non-response bias) it should be evaluated carefully before being implemented. In 
the following sections, each of these concerns is addressed. 

Comparative Costs of Each Design 

Table 3 contains data on the estimated costs of each design and the relative value of each. These 
costs reflect only the wages paid to the staff to prepare the mailings and the postage costs. The wages are 
further limited to only the time required for printing individually addressed letters on a laser printer, signing 
the letters, printing envelopes and postcards on a dot-matrix printer with an envelope feeder, preparing the 
mailing packages, and checking the accuracy of the names, addresses, and control numbers. These costs 
were calculated for the first survey and the costs for the other two survey costs were standardized on the 
first survey. 

The data in Table 3 permit an evaluation of both the sponsor letter and the postcard. From these 
data, it appears that the sponsor letter did not produce responses equal to their costs. For example, the 
costs of two or three mailings without the sponsor letter was less than $3.00 per response. When the 
sponsor letter was sent as either a follow-up after two or three mailings or as a presurvey letter, the costs 
increased substantially. The third survey and the first survey had an equivalent number of contacts. The 
results further indicate that the presurvey letter is relatively costly and does not produce as many returns 
(at a lower unit cost) as mailing a third questionnaire. 
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Table 3: Costs per Response With Various Mail Survey Design Procedures. 


PROCEDURES AND COSTS 

NASA1 1 

NASA2 2 

NASA3 3 

Cost per response prior 

to sponsor letter 

$2.92 

$2.96 


Response Rate 

58% 

50% 


Adjusted Response Rate 

59% 

52% 


Return Rate 

61% 

55% 


Cost per response - 

sponsor letter only 

$8.41 

$6.04 


Cost per response with 

a sponsor letter 

$3.21 

$3.27 

$3.12 

Response Rate 

61% 

56% 

54% 

Adjusted Response Rate 

67% 

63% 

64% 

Return Rate 

75% 

72% 

74% 


1 . Initial mailing and postcard sent to all respondents; up to two additional questionnaires sent to non- 
respondents, sponsor letter sent last. 

2. Initial mailing and postcard sent to all respondents; one additional questionnaire sent to non- 
respondents, sponsor letter sent last. 

3. Presurvey sponsor letter, mailing and postcard to all who did not indicate not applicable, one 
additional questionnaire sent to non-respondents. 


If the Adjusted Response Rate is used to evaluate the procedures, then the postcard does seem to 
be cost-effective. In the analysis of Table 2, it was demonstrated that the postcard helped to refine the 
sample by improving the Adjusted Response Rate. Each of the surveys has about the same Adjusted 
Response Rate but the third survey cost somewhat less than the other two. 

Overall, the results indicate that a sponsor letter as a follow-up to a survey is not cost effective; 
especially since the cost per return of the extra mailing was over $8.00 and $6.00 in the two surveys. The 
data did not support our assumption that a presurvey letter with a postcard would reduce costs. We 
assumed that ineligible or non-cooperative respondents would be eliminated and this would reduce costs 
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because of fewer mailings. The postcard was cost effective in clarifying the sample; especially when it was 
sent with a presurvey letter. 

Impact of the Procedures on Non-Response Bias 

The list of AIAA members that comprises the frame for the sample includes a number of 
characteristics that were also measured in the questionnaire. Two characteristics - education and 
government /non-government employment - are relevant to the focus of the research. Ideally, there should 
be no difference between the distributions of these characteristics in the frame and in the data from each of 
the surveys. We did suspect that the sample would be biased on these two characteristics. First, scientists 
and engineers with advanced degrees would be more likely to be engaged in aerospace research and 
therefore more likely to find the questionnaires to be relevant. Second, we expected that those employed 
in government agencies would be more likely to return the questionnaires because they were more likely to 
be involved in aerospace research. 

The data in Table 4 support our hypotheses. First, there is clear evidence that the a larger 
proportion of respondents to each of the surveys has advanced degrees than in the sample frame in 
general. In the sample frame only 57 percent have advanced degrees but in the surveys at least 66 percent 
reported they have a master’s degree or a doctorate. 

The academic portion of the survey was conducted separately so those classified as non- 
government in employment question are primarily working in industry. The relatively higher proportion of 
government agency employees may reflect differences in professional duties. That is, we expect that 
members of the AIAA who are employed in government settings are more likely to be engaged in 
aerospace research. Many of the professional duties of members of the AIAA who are in industry, e.g., 
sales and service, are not found in government. 
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Table 4: Characteristics of the Sample Frame and the Respondents to Three Surveys of 

Members of AAIA. (percents) 

SAMPLE NASAl NASA2 NASA3 

FRAME 

CHARACTERISTICS 


Education 

Bachelor’s 36 

Master’s 36 

Doctorate 21 

Other 7 

Employment 

Non-Government 80 

Government 20 


31 

31 

29 

43 

45 

44 

23 

22 

24 

3 

2 

3 


74 

74 

77 

26 

26 

23 


These data may indicate a non-response bias but they can also be interpreted in other ways. First, 
we are not certain that the distribution of responses to any questions on the survey would be different if we 
had received questionnaires from all persons in the sample. Second, the non-respondents may not be 
actively engaged in aerospace research. If so, then if the non-respondents did return questionnaires, it 
would create another form of bias. That is, they would provide answers even though the questions were 
not relevant. It is possible that questionnaires were received from almost all persons to whom the focus of 
the research was relevant. Overall, it is not clear what impact the procedures had on errors associated with 
non-response to the surveys. 

Summary 

In this paper we described the impact of two interventions (a sponsor letter and a postcard) on 
improvements to response rates and the clarification of a sample. We evaluated the effectiveness by an 
increase in the response rates, the relative costs of the procedures, and the impact of the procedures on 
the characteristics of the respondents. The results indicate that the sponsor letter had little impact on the 
results. The increased cost of mailing a separate letter from a sponsor did not result in as many responses 
as would another mailing of the questionnaire. 
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The postcard appears to be an effective means of clarifying the appropriate sample when there is a 
sample frame that may contain persons who are not eligible for the survey. The postcard was especially 
effective when it was included with a presurvey letter. In addition, the postcard sent with the presurvey 
letter was cost effective when compared to mailing the postcard after the survey had begun. 

None of the three mail survey designs produced differences in the characteristics of the sample. 
Likewise, all designs showed evidence of non-response bias. It is not possible to estimate the impact of 
the nonresponse bias but it can be argued that the non-respondents may not have been eligible for the 
survey. 
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